Learning Visual Routines with Reinforcement Learning

Reinforcement learning is an ideal framework to learn visual routines since the routines are made up of sequences of actions. However, such algorithms must be able to handle the hidden state (perceptual aliasing) that results from visual routine’s purposefully narrowed attention. The U-Tree algorithm successfully learns visual routines for a complex driving task in which the agent makes eye movements and executes deictic actions in order to weave in and out of traffic on a four-laned highway. The task involves hidden state, time pressure, stochasticity, a large world state space, and a large perceptual state space. U-Tree uses a tree-structured representation, and is related to work on Prediction Suffix Trees (Ron, Singer, &; Tishby 1994), Parti-game (Moore 1993), algorithm (Chapman 8z Kaelbling 1991), and Variable Resolution Dynamic Programming (Moore 1991). Tree is a direct descendant of Utile Suffix Memory (McCallum 1995c), which used short-term memory, but not selective perception. Unlike Whitehead’s Lion algorithm, the algorithm handles noise, large state spaces, and uses short-term memory to uncover hidden state.